Collect
Improve Data Extraction Efficiency
- Dedicated hardware – The fewer applications and users sharing the same hardware, the better and more consistent the performance.
- High speed connectivity – Bandwidth is often a bottleneck.
- Extract only relevant tables – Clearly define the tables that are in scope for extraction.
- Extract only relevant records/and or columns from tables – Excluding fields that are not being used effectively makes a narrower table. In some cases, the client may request that certain sensitive fields be excluded from extraction altogether. Adding relevancy (for example, pulling only active customers) also reduces the amount of data being extracted.
- Create a development environment with comparable performance to production environment – The testing and configuration process is critical. Testing in an environment that closely mimics that of development provides a much more accurate picture of performance and behavior.
- Baseline and monitor extraction performance – Variables that impact efficiency change over time and it is important to monitor performance. Such variables include record volumes, changes in hardware or software, changes in business processes or protocols, and load on the network.
